Apparatus and method for producing and storing multiple video streams

ABSTRACT

A digital imaging device produces and stores multiple video streams from a single imaging sensor array. The stored video streams may be post-processed using video editing software to create a combined video presentation.

FIELD OF THE INVENTION

[0001] The present invention relates generally to digital imaging andmore specifically to digital still cameras and digital camcorders.

BACKGROUND OF THE INVENTION

[0002] Both digital camcorders and digital still cameras having a videomode produce a single digital video stream that may be readily editedand, if desired, stored on an optical medium such as a DVD. Some devicesare also capable of operating in picture-in-picture (PIP) mode, in whichan inset second view is embedded within a larger background view. Insome devices, it is even possible to record video in this PIP mode.Thus, two different views of the scene are recorded simultaneously in asingle video stream.

[0003] In some applications, however, it is advantageous to generate twoor more different views as separate, independent video streams.Typically, this is accomplished through the use of multiple cameras.Using multiple cameras can be both expensive and cumbersome.

[0004] It is thus apparent that there is a need in the art for animproved apparatus and method for producing and storing multiple videostreams.

SUMMARY OF THE INVENTION

[0005] A digital imaging device that produces and stores multiple videostreams is provided. An associated method for capturing digital video inthe digital imaging device is also provided.

[0006] Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007]FIGS. 1A and 1B are functional block diagrams of a digital imagingdevice in accordance with an illustrative embodiment of the invention.

[0008]FIG. 2A is an illustration showing disjoint first and secondregions of an imaging sensor array in accordance with an illustrativeembodiment of the invention.

[0009]FIG. 2B is an illustration showing a first example of overlappingfirst and second regions of an imaging sensor array in accordance withan illustrative embodiment of the invention.

[0010]FIG. 2C is an illustration showing a second example of overlappingfirst and second regions of an imaging sensor array in accordance withan illustrative embodiment of the invention.

[0011]FIG. 3A is an illustration of sampling overlapping first andsecond regions of a CMOS imaging sensor array in accordance with anillustrative embodiment of the invention.

[0012]FIG. 3B is an illustration of sampling overlapping first andsecond regions of a CCD imaging sensor array in accordance with anillustrative embodiment of the invention.

[0013]FIG. 4 is a flowchart of the operation of the digital imagingdevice shown in FIGS. 1A and 1B in accordance with an illustrativeembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0014] Multiple video streams may be produced simultaneously in adigital imaging device by sampling multiple regions of a single imagingsensor array. Saving the multiple video streams in a memory provides avariety of post-production options. For example, portions of the videostreams may be edited into a single combined video presentation, or twoor more video streams may be combined in split-screen,picture-in-picture, or auto-fade fashion.

[0015] In a specific example, a parent records a child's piano recitalusing a digital camcorder. The digital camcorder produces and recordstwo independent video streams simultaneously: (1) a wide-angle viewshowing the child playing the piano from 15 feet away and (2) a croppedview showing only the child's torso and hands at the keyboard. Since thetwo video streams are saved separately, video editing software may beused to create a single video presentation combining portions of boththe wide-angle and close-up views for a more varied, interestingpresentation. In some embodiments, this editing and combining of the twovideo streams may be performed manually using the video editingsoftware. In other embodiments, the video editing software mayautomatically combine the two streams to create a single video stream.In automatically combining the two video streams, the video editingsoftware may employ techniques well known in the art such as picture inpicture, split screen, or auto-fade. In the case of auto-fade, forexample, the video editing software may alternately fade between thefirst and second video streams at predetermined intervals.

[0016]FIGS. 1A and 1B are functional block diagrams of a digital imagingdevice 100 in accordance with an illustrative embodiment of theinvention. Digital imaging device 100 may be a digital still camera withthe capability of operating in video mode, a digital camcorder, or anysimilar device capable of capturing digital video. In FIG. 1A,controller 105 communicates over data bus 110 with imaging module 115,memory 120, and display 125. Optical system 130 produces optical imagesthat are converted to digital images by imaging module 115. Memory 120may further comprise random access memory (RAM) 135 and non-volatilememory 140. Non-volatile memory 140 may be a CompactFlash™, SmartMedia™, or Secure Digital™ card; magnetic tape; magnetic disk; oroptical disc. As shown in FIG. 1B, imaging module 115 may furthercomprise an imaging sensor array 145 based on charge-coupled-device(CCD) or CMOS technology and sampling logic 150. Sampling logic 150controls the sequence and timing of reading (sampling) imaging sensorarray 145 to produce digital video. Imaging module 115 typically alsoincludes an analog-to-digital converter (A/D), a gain control, and adigital signal processor (DSP), as is well known in the art (not shownin FIG. 1B). Display 125 may, among other things, provide a preview ofthe scene currently emanating from optical system 130 prior torecording, a view of what is being recorded during recording, and anon-screen user interface for controlling the operating modes and optionsof digital imaging device 100.

[0017]FIGS. 2A-2C are illustrations showing examples of ways in which afirst region 205 and a second region 210 of a single imaging sensorarray 145 may be sampled to produce two independent video streams thatmay be stored in memory 120. Two regions (205 and 210) are shown inFIGS. 2A-2C for simplicity. In other embodiments, more than two regionsof imaging sensor array 145 may be sampled to produce more than twoindependent video streams. First region 205 may be sampled at a firstresolution R1; second region 210, at a second resolution R2. R1 and R2may be different or the same, depending on the application.

[0018]FIG. 2A shows an embodiment in which first region 205 and secondregion 210 are disjoint. FIG. 2B shows another embodiment in which firstregion 205 comprises a central portion of imaging sensor array 145 andsecond region 210 comprises a background view that spans the fulldimensions of imaging sensor array 145. FIG. 2C shows yet anotherembodiment in which first region 205 and second region 210 partiallyoverlap.

[0019]FIGS. 2A-2C are merely illustrative. A variety of otherconfigurations are possible in which both a first region 205 and asecond region 210 of imaging sensor array 145 are sampled and theresulting video streams are stored in memory 120.

[0020] Implementation of the invention may differ somewhat depending onwhether imaging sensor array 145 is of the CCD or CMOS type. Oneadvantage of a CMOS imaging sensor array is that sampling logic 150 mayaddress it directly at the picture element (pixel) level, much like aRAM. However, some CMOS imaging sensor arrays may be read at a givenpixel only once per exposure (destructive read).

[0021]FIG. 3A is an illustration showing how first and second regions205 and 210, respectively, may be sampled from a CMOS imaging sensorarray 145 in a situation in which the two regions partially overlap. Theportion of imaging sensor array 145 marked with “X's” may be ignored(not sampled). Within first region 205, imaging sensor array 145 may besampled at resolution R1. Within second region 210, imaging sensor array145 may be sampled at resolution R2. In the case of a destructive read,additional care must be taken with the overlapping portion of the tworegions 205 and 210. In the example shown in FIG. 3A, R1 is assumed tobe greater than R2. As those skilled in the art will recognize, adestructive read necessitates downsampling or scaling, in memory, theoverlapping portion of first region 205 to produce the remainder of thevideo stream associated with second region 210 at (lower) resolution R2.In general, in the case of a destructive read and overlapping regions,the region to be sampled at higher resolution should be sampled beforethe region to be sampled at lower resolution.

[0022]FIG. 3B illustrates how the sampling of first region 205 andsecond region 210 differs if imaging sensor array 145 is a CCD. SinceCCD's are typically sampled by shifting a row of sensor elements at atime into a set of horizontal shift registers (not shown in FIG. 3B) andthen reading out a row of pixels in succession from the shift registers,imaging sensor array 145 may be logically divided into “strips,” asshown in FIG. 3B. In this example, it is assumed that the rows of pixelsare shifted downward in FIG. 3B into the set of horizontal shiftregisters. The top and bottom strips marked with “X's” are shifted intothe set of horizontal shift registers but are skipped (not read intomemory by sampling logic 150). The strip beside which “R1” has beenplaced in FIG. 3B may be read out at resolution R1; the strip besidewhich “2” has been placed, at resolution R2, where R1 is greater thanR2. The portion of the strip read out at R1 and containing a portion ofsecond region 210 may be downsampled to produce the remainder of thevideo stream associated with second region 210, as explained above inconnection with CMOS imaging sensor arrays. The pixels lying outsidefirst region 205 and second region 210 in any given row of pixels may beskipped when pixels from the applicable row are read from the set ofhorizontal shift registers.

[0023]FIG. 4 is a flowchart of the operation of the digital imagingdevice 100 shown in FIGS. 1A and 1B in accordance with an illustrativeembodiment of the invention. At 405, sampling logic 150 samples firstregion 205 of imaging sensor array 145 at a first resolution to producea first video stream. At 410, sampling logic 150 samples second region210 of imaging sensor array 145 at a second resolution to produce asecond video stream. As mentioned previously, the first and secondresolutions may, in some embodiments, be the same. At 415, the first andsecond video streams are stored in memory 120, which may, depending onthe embodiment, be RAM 135 or non-volatile memory 140, or both. Atoptional step 420, the first and second video streams may bepost-processed using suitable video editing software, as explainedabove. The process terminates at 425.

[0024] In some embodiments, the first and second video streams may, at415, be stored in a single data file organized such that the two videostreams are separable (separately readable). In other embodiments, thetwo video streams may be stored in separate data files at 415. In eithercase, the resulting video streams may be post-processed using suitablevideo editing software.

[0025] The foregoing description of the present invention has beenpresented for the purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed, and other modifications and variations may be possible inlight of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical application to thereby enable others skilled in the art tobest utilize the invention in various embodiments and variousmodifications as are suited to the particular use contemplated. It isintended that the appended claims be construed to include otheralternative embodiments of the invention except insofar as limited bythe prior art.

What is claimed is:
 1. A digital imaging device, comprising: an imagingsensor array to capture digital video; sampling logic configured tosample a first region of the imaging sensor array at a first resolutionto produce a first video stream and to sample a second region of theimaging sensor array at a second resolution to produce a second videostream, the first and second regions being different; and a memory tostore the first and second video streams.
 2. The digital imaging deviceof claim 1, wherein the first resolution and the second resolution arethe same.
 3. The digital imaging device of claim 1, wherein the firstregion and the second region are disjoint.
 4. The digital imaging deviceof claim 1, wherein the first region and the second region partiallyoverlap.
 5. The digital imaging device of claim 1, wherein the first andsecond video streams occupy separate files in the memory.
 6. The digitalimaging device of claim 1, wherein the first and second video streamsoccupy a single file in the memory.
 7. The digital imaging device ofclaim 1, wherein the imaging sensor array comprises a charged-coupleddevice.
 8. The digital imaging device of claim 1, wherein the imagingsensor array comprises a CMOS device.
 9. The digital imaging device ofclaim 1, wherein the digital imaging device comprises a digital stillcamera capable of operating in a digital video mode.
 10. The digitalimaging device of claim 1, wherein the digital imaging device comprisesa digital camcorder.
 11. The digital imaging device of claim 1, whereinthe memory comprises at least one of a magnetic tape, a magnetic disk,an optical disc, a random access memory, and a flash memory.
 12. Amethod for capturing digital video in a digital imaging device,comprising: sampling a first region of an imaging sensor array at afirst resolution to produce a first video stream; sampling a secondregion of the imaging sensor array at a second resolution to produce asecond video stream, the first and second regions being different; andstoring the first and second video streams in a memory.
 13. The methodof claim 12, wherein the first resolution and the second resolution arethe same.
 14. The method of claim 12, wherein the first region and thesecond region are disjoint.
 15. The method of claim 12, wherein thefirst region and the second region partially overlap.
 16. The method ofclaim 12, wherein the first and second video streams are stored inseparate files.
 17. The method of claim 12, wherein the first and secondvideo streams are stored in a single file.
 18. The method of claim 12,wherein the digital imaging device comprises a digital still cameracapable of operating in a digital video mode.
 19. The method of claim12, wherein the digital imaging device comprises a digital camcorder.20. The method of claim 12, further comprising: post-processing thefirst and second video streams using video editing software.
 21. Themethod of claim 20, wherein the video editing software automaticallycombines the first and second video streams to produce a third videostream.
 22. The method of claim 21, wherein the third video streamincorporates at least one of picture in picture, split screen, and autofade
 23. A digital imaging device, comprising: means for capturingdigital video; means for sampling a first region of the means forcapturing digital video at a first resolution to produce a first videostream and for sampling a second region of the means for capturingdigital video at a second resolution to produce a second video stream,the first and second regions being different; and means for storing thefirst and second video streams.